Display system and server

ABSTRACT

A display system includes a display device and a remote terminal device. The display device includes an image shooting device, an image recognizer, and a display controller. The image recognizer recognizes real objects in an image shot by the image shooting device, and recognizes positions of the recognized real objects on space coordinates. The display controller superimposes a virtual object at a designated position on the space coordinates in the shot image. The image recognizer recognizes a gesture of the real object at the designated position of the virtual object in the shot image. The display controller executes image processing on the virtual object in response to the recognized gesture.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Japanese Patent Application No.2020-211009 filed on Dec. 21, 2020, incorporated herein by reference inits entirety.

BACKGROUND 1. Technical Field

Disclosed herein are a display system configured to display an augmentedreality (AR) image, and a server in the display system.

2. Description of Related Art

Hitherto, there is known a display device using an augmented realitytechnology. For example, in Japanese Unexamined Patent ApplicationPublication No. 2019-075125 (JP 2019-075125 A), image processing isexecuted on a virtual object by making a gesture on the virtual object.For example, a folder icon (storage icon) is projected as the virtualobject on a head-mounted display (HMD). For example, image processingfor opening the folder icon is executed by making a gesture such aspinching of the three-dimensional (3D) icon with fingers.

SUMMARY

A service of a so-called AR game such as a game of chasing and capturinga fleeing virtual object may be provided by using the augmented realitytechnology. If the fleeing route of the fleer virtual object is providedin the same pattern, it is difficult to keep the attractiveness of thegame. If an algorithm is created to complicate the fleeing route, asystem that provides the AR game may have an excessive load.

Disclosed herein are a display system and a server in the displaysystem, in which the attractiveness of a provided AR game can be keptwhile reducing a load on a system that provides the game.

A display system is disclosed herein. The display system includes adisplay device and a manipulation device. The display device includes animage shooting device, an image recognizer, and a display controller.The image shooting device is configured to shoot a scene in a realworld. The image recognizer is configured to recognize real objects inan image shot by the image shooting device, and recognize positions ofthe recognized real objects on space coordinates. The display controlleris configured to superimpose a virtual object at a designated positionon the space coordinates in the shot image. The image recognizer isconfigured to recognize a gesture of the real object at the designatedposition of the virtual object in the shot image. The display controlleris configured to execute image processing on the virtual object inresponse to the recognized gesture. The manipulation device includes aninputter configured to input a manipulation command to change thedesignated position of the virtual object.

According to the configuration described above, the virtual objectserving as, for example, a chasing target in an AR game is manipulatedby a manipulator who inputs the manipulation command to the manipulationdevice. Thus, it is possible to avoid, without creating a complicatedalgorithm, a case where the fleeing route of the virtual object isprovided in the same pattern.

In the configuration described above, the display system may furtherinclude a server communicable with the display device and themanipulation device. In this case, the display device may be arranged ina complex facility including a plurality of facilities with differentspecific themes. The server may include a storage and a transmitter. Thestorage may be configured to store character images set as characters inthe individual facilities in association with pieces of positionalinformation in the complex facility. The transmitter may be configuredto transmit data on the character images to the display device togetherwith the pieces of positional information associated with the characterimages. The display controller may be configured to select, as an imageof the virtual object, the character image associated with thedesignated position, and superimpose the character image on the shotimage.

According to the configuration described above, the character imageharmonized with a world of the facility such as a theme park can bedisplayed as the image of the virtual object.

In the configuration described above, the gesture may be a capturingaction at the designated position. In this case, the display device maybe configured to transmit a capture notification to the server when theimage recognizer recognizes the capturing action performed by the realobject. The server may include a reward calculator configured todetermine a reward for a manipulator of the manipulation device based ona fleeing period from a start timing of manipulation of the virtualobject to a timing of recognition of the capturing action.

According to the configuration described above, the reward is given tothe manipulator of the virtual object as a motivation to manipulate thevirtual object while spending time.

In the configuration described above, the display device may include aninputter configured to input participation registration as a chaser in achasing game for chasing the virtual object. In this case, themanipulation device may include an inputter configured to inputparticipation registration in the chasing game as a manipulator of thevirtual object. The server may include a participant calculatorconfigured to calculate the numbers of the chasers and the manipulatorsparticipating in the chasing game, and acquire positions in the complexfacility about the virtual object manipulated by the manipulator and thedisplay device of the chaser. The reward calculator may be configured todetermine the reward for the manipulator based on a virtual object spacedensity around the captured virtual object at the timing of recognitionof the capturing action.

According to the configuration described above, different rewards can begiven, for example, between a state in which the virtual object spacedensity is high, that is, the capture is difficult and a state in whichthe virtual object space density is low, that is, the capture is easy.

In the configuration described above, the reward calculator may beconfigured to determine the reward for the manipulator of themanipulation device based on a chaser space density around the capturedvirtual object at the timing of recognition of the capturing action.

According to the configuration described above, different rewards can begiven between a state in which the chaser space density is high, thatis, the capture is easy and a state in which the chaser space density islow, that is, the capture is difficult.

A server is disclosed herein. The server is communicable with displaydevices and manipulation devices. Each display device is configured todisplay a virtual object in a shot image obtained by shooting a scene ina real world, and execute a chasing game for chasing the virtual object.Each manipulation device includes an inputter configured to input amanipulation command for the virtual object. The server includes astorage and a transmitter. The storage is configured to store pieces ofpositional information of the display devices participating in thechasing game, and pieces of positional information of the virtualobjects manipulated by the manipulation devices. The transmitter isconfigured to transmit, to a predetermined display device among thedisplay devices, the positional information of the virtual object aroundthe display device based on the positional information of the displaydevice.

According to the display system and the server disclosed herein, theattractiveness of the AR game can be kept while reducing the load on thesystem that provides the AR game.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, advantages, and technical and industrial significance ofexemplary embodiments of the present disclosure will be described belowwith reference to the accompanying drawings, in which like signs denotelike elements, and wherein:

FIG. 1 is a diagram exemplifying an entertainment complex and a house ata remote place using a display system according to an embodiment;

FIG. 2 is a diagram for describing character display areas in theentertainment complex;

FIG. 3 is a diagram exemplifying the hardware configurations of devicesin the display system according to the embodiment;

FIG. 4 is a diagram exemplifying functional blocks of a server accordingto the embodiment;

FIG. 5 is a diagram illustrating a smartphone as an example of a displaydevice;

FIG. 6 is a diagram exemplifying functional blocks of the display deviceaccording to the embodiment;

FIG. 7 is a diagram exemplifying functional blocks of a remote terminaldevice serving as a manipulation device according to the embodiment;

FIG. 8 is a diagram exemplifying a chaser flow in a chasing game usingthe display system according to the embodiment;

FIG. 9 is a diagram exemplifying a scene in the entertainment complex;

FIG. 10 is a diagram exemplifying a shot image obtained by shooting thescene of FIG. 9 by using the display device;

FIG. 11 is a diagram exemplifying a process of capturing a virtualobject (1/2);

FIG. 12 is a diagram exemplifying the process of capturing the virtualobject (2/2);

FIG. 13 is a diagram exemplifying a fleer flow in the chasing game usingthe display system according to the embodiment;

FIG. 14 is a diagram exemplifying a game image displayed on a displayunit of the remote terminal device;

FIG. 15 is a diagram exemplifying functional blocks of a serveraccording to another example of the embodiment;

FIG. 16 is a diagram exemplifying functional blocks of a display deviceaccording to the other example of the embodiment;

FIG. 17 is a diagram exemplifying functional blocks of a remote terminaldevice serving as the manipulation device according to the other exampleof the embodiment;

FIG. 18 is a diagram exemplifying a chaser flow in a chasing game usinga display system according to the other example of the embodiment;

FIG. 19 is a diagram exemplifying a fleer flow in the chasing game usingthe display system according to the other example of the embodiment; and

FIG. 20 is a diagram exemplifying a head-mounted display as anotherexample of the display device according to the embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

FIG. 1 exemplifies an entertainment complex 10 and a house 92 using adisplay system according to an embodiment. The house 92 is located at aplace remote from the entertainment complex 10, and a manipulator 94 whouses the display system is in the house 92. The display system includesa display device 30 and a server 70 provided in the entertainmentcomplex 10, and a remote terminal device 100 (manipulation device)provided in the house 92.

Configuration of Entertainment Complex

The entertainment complex 10 includes a plurality of theme parks 14, 16,and 18. The theme park is a field where a world is created based on aspecific theme (subject) and facilities, events, and scenery areelaborated integrally based on the world. For example, the theme parks14, 16, and 18 are connected by access roads 20, and visitors to theentertainment complex 10 can move around the theme parks 14, 16, and 18via the access roads 20.

The entertainment complex 10 includes theme parks having differentthemes. For example, the entertainment complex 10 includes an amusementpark 14, an aquarium 16, and a zoo 18 as the theme parks.

Characters are set based on the themes of the individual theme parks 14,16, and 18. The characters are set in harmony with the themes and worldsof the theme parks 14, 16, and 18. Examples of characters in theamusement park 14 include a clown and a go-kart. Examples of charactersin the aquarium 16 include a fish and a shark. Examples of characters inthe zoo 18 include a monkey, a lion, and a panda.

As described later, images of those characters (hereinafter referred toas “character images” as appropriate) are used as images of fleers in achasing game that is an AR game service available in the entertainmentcomplex 10.

The chasing game can be played by visitors to the entertainment complex10, and is typified by “tag” using an augmented reality technology. Inthe chasing game, a virtual image of a fleer serving as a chasing targetis superimposed on a shot image showing the inside of the entertainmentcomplex 10 and the superimposed image (AR image) is displayed on thedisplay device 30 as described later. A visitor acting as a chaserchases the virtual image to capture the virtual image. In the displaysystem according to this embodiment, the virtual image of the fleer ismanipulated by using the remote terminal device 100 remote from theentertainment complex 10.

The visitors can move not only in the theme parks 14, 16, and 18 andalong the access roads 20 but also in any area inside the entertainmentcomplex 10. For example, areas outside the access roads 20 aregrasslands where the visitors can stroll. In the chasing game, achaseable area (chasing field) is set to, for example, the entire areain the entertainment complex 10. In the chasing game, the area in theentertainment complex 10 is virtually divided into a plurality of areasas exemplified in FIG. 2. For example, the area is divided into threeareas that are an amusement park area 14A including the amusement park14, an aquarium area 16A including the aquarium 16, and a zoo area 18Aincluding the zoo 18. Borders among the areas are virtual, and nophysical borders such as a fence are provided. The visitors to theentertainment complex 10 can freely move to the individual areas.

In the chasing game, images of virtual objects (character images) to bedisplayed in the areas 14A to 18A are set different from each other. Forexample, a character image set for the amusement park 14 is displayed asa virtual object image in the amusement park area 14A. Similarly, acharacter image set for the aquarium 16 is displayed as a virtual objectimage in the aquarium area 16A. A character image set for the zoo 18 isdisplayed as a virtual object image in the zoo area 18A.

Pieces of data on the character images (character image data) in thetheme parks 14, 16, and 18 are stored in a theme park-specific characterstorage 82 of the server 70 (see FIG. 4) in association withidentification symbols and pieces of positional information of the themeparks 14, 16, and 18. For example, the server 70 is installed in amanager building (not illustrated) in the entertainment complex 10.Details of the character image data stored in the theme park-specificcharacter storage 82 are described later.

Referring back to FIG. 1, the entertainment complex 10 includes beacontransmitters 22. For example, a plurality of transmitters 22 is providedat regular intervals. As described later, a beacon receiver 37 of thedisplay device 30 (see FIG. 3) receives a signal from any transmitter 22to acquire a current position of the display device 30.

The remote terminal device 100 is arranged in the house 92 at the placeremote from the entertainment complex 10. As described later in detail,a fleer virtual object (character) in the chasing game serving as the ARgame in the display system according to this embodiment is manipulatedby using the remote terminal device 100. That is, a visitor to theentertainment complex 10 participates in the chasing game as a chaser,and the virtual object serving as a chasing target is manipulated suchthat the manipulator 94 remote from the entertainment complex 10 inputsa manipulation command to the remote terminal device 100.

Configurations of Devices in Display System

FIG. 3 exemplifies the hardware configurations of devices in the displaysystem according to this embodiment. The display system according tothis embodiment includes the display device 30, the server 70, and theremote terminal device 100.

Configuration of Server

Referring to FIG. 3, the server 70 is, for example, a computer andinstalled in the management building in the entertainment complex 10(see FIG. 1). The server 70 is wirelessly connected to the displaydevice 30 by a communication method such as a wireless local areanetwork (LAN). The server 70 is communicable with the remote terminaldevice 100 by a communication method such as the Internet.

The server 70 includes an inputter 71 including a keyboard and a mouse,a central processing unit (CPU) 72 serving as a processor, and a displayunit 73 such as a display. The server 70 includes a read-only memory(ROM) 74, a random-access memory (RAM) 75, and a hard disk drive (HDD)76 as storages. The server 70 includes an input/output controller 77configured to manage input and output of information. Those componentsare connected to an internal bus 78.

FIG. 4 exemplifies functional blocks of the server 70. For example, thefunctional blocks are implemented such that the CPU 72 executes programsstored in the ROM 74, the HDD 76, or a non-transitory computer-readablestorage medium such as a digital versatile disc (DVD).

The server 70 includes a receiver 83, a participant calculator 84, areward calculator 85, a 3D model extractor 86, and a transmitter 87 asfunctional units. The server 70 includes a chasing game participantstorage 80, a facility map storage 81, and the theme park-specificcharacter storage 82 as storages.

The chasing game participant storage 80 stores participant informationin the chasing game serving as the AR game. Participants include chasersand fleers. The chasers include any visitor to the entertainment complex10 who has sent a request for participation registration as a chaser inthe chasing game from the display device 30 of the visitor. The fleersinclude a person who has sent a request for participation registrationas a fleer in the chasing game from the remote terminal device 100.

The chasing game participant storage 80 stores pieces of information ona chaser and a fleer participating in the chasing game. For example,chaser information includes an identification symbol and positionalinformation of the display device 30 of the chaser. Fleer informationincludes an identification symbol of the remote terminal device 100manipulated by the manipulator 94 (see FIG. 1), and positionalinformation of a virtual object manipulated by using the remote terminaldevice 100.

The facility map storage 81 stores park map data that is map informationin the entertainment complex 10. For example, pieces of positionalinformation of the access roads 20 and the facilities in theentertainment complex 10 are stored as the park map data. Further,positional information of a no-entry area such as a pond 12 (see FIG. 1)is stored. For example, the no-entry area is set as a fleer's movementprohibition area.

Specifically, the facility map storage 81 stores map data of theentertainment complex 10 as the park map data in association withpositional information. The positional information is positionalinformation in a so-called world coordinate system. The positionalinformation in the world coordinate system includes information on alatitude, a longitude, and an altitude acquirable by using a globalpositioning system (GPS) function, and space coordinate informationacquirable by using a beacon function.

The facility map storage 81 stores pieces of data on the areas 14A, 16A,and 18A in the chasing game in FIG. 2. For example, border lines of theareas 14A, 16A, and 18A are set in the park map data, and identificationsymbols of the areas 14A, 16A, and 18A are assigned to areas defined bythe border lines.

The theme park-specific character storage 82 stores pieces of data oncharacter images that are fleer images to be displayed as virtual objectimages on the display device 30. The characters are set as thecharacters in the theme parks 14, 16, and 18 in harmony with theirworlds. For example, a plurality of types of character image data isstored in the theme park-specific character storage 82 for each of thetheme parks 14, 16, and 18. For example, 10 types or more and 100 typesor less are stored in the theme park-specific character storage 82 asthe character image data of one theme park. The pieces of characterimage data have identification symbols for each of the theme parks 14,16, and 18, that is, the areas 14A, 16A, and 18A. Each piece ofcharacter image data has a unique identification symbol.

The character image data serving as the virtual object is stored in thetheme park-specific character storage 82 together with positionalinformation associated with the data. The positional informationassociated with the character image data includes information on an areain the entertainment complex 10 where display of the image data ispermitted. The character image data and the associated positionalinformation data are hereinafter referred to collectively as “virtualobject data” as appropriate.

As exemplified in FIG. 2, character images 15 whose identificationsymbols are associated with the amusement park 14 and whose displayareas are set to the amusement park area 14A include an image of a clownbalancing on a ball. Character images 17 whose identification symbolsare associated with the aquarium 16 and whose display areas are set tothe aquarium area 16A include an image of a swimming fish. Similarly,character images 19 whose identification symbols are associated with thezoo 18 and whose display areas are set to the zoo area 18A include animage of a running monkey.

The character image data may be 3D model data. For example, the 3D modeldata includes 3D image data of the character. The image data includesshape data, texture data, and motion data. The image data also includesdata on an effect image to be displayed when the virtual object iscaptured.

For clarity, FIG. 2 illustrates outlines of the character images 15, 17,and 19. The character images 15, 17, and 19 are not limited to theoutlines, and 3D images may be displayed as the character images 15, 17,and 19.

For example, the participant calculator 84 calculates the number ofparticipants in the chasing game. The participants include a chaser anda fleer. The participant calculator 84 acquires positional informationin the entertainment complex 10 about the chaser participating in thechasing game, more specifically, the display device 30 of the chaser.The participant calculator 84 acquires positional information in theentertainment complex 10 about the fleer (virtual object) manipulated bya manipulator participating in the chasing game. For example, theparticipant calculator 84 calculates a chaser space density and a fleerspace density (virtual object space density) described later byreferring to the chasing game participant storage 80.

Based on positional information of the display device 30 participatingin the chasing game (as a chaser), the participant calculator 84provides positional information of a virtual object (fleer) around theposition to the display device 30. The participant calculator 84provides the park map data and the pieces of positional information ofthe fleer and the chaser to the remote terminal device 100 participatingin the chasing game (as the fleer).

The 3D model extractor 86 extracts character image data of a virtualobject from the theme park-specific character storage 82. The extractedimage data is transmitted to the display device 30 and the remoteterminal device 100 via the transmitter 87.

The 3D model extractor 86 may transmit pieces of character image data inall the areas 14A, 16A, and 18A to the display device 30 and the remoteterminal device 100. For example, one arbitrary type of character imagedata in each of the areas 14A, 16A, and 18A is transmitted to thedisplay device 30 and the remote terminal device 100.

The reward calculator 85 calculates a reward to be given to amanipulator who has manipulated a fleer in the chasing game. Forexample, the reward is calculated based on a fleeing period. Otherelements for the reward calculation include a fleer space density and achaser space density on the periphery when the fleer is captured.Examples of the reward include a discount service for an admission feeof the entertainment complex 10. The reward calculation is describedlater.

Configuration of Display Device 30

Referring to FIG. 1, the display device 30 is provided in theentertainment complex 10, and is used by a visitor (user) to theentertainment complex 10. The display device 30 can display a virtualreality image by superimposing an image of a virtual object on a scenein the real world.

The display device 30 may be a mobile device. Examples of the displaydevice 30 include a smartphone including an image shooting device and adisplay unit, and a head-mounted display (HMD) in the form of glasses.

In view of methods for displaying scenes in the real world, the displaydevice 30 may be categorized into a video see-through display (VSTdisplay) and an optical see-through display (OST display). In the VSTdisplay, an image shooting device such as a camera shoots a scene in thereal world, and a display unit displays the shot image. In the OSTdisplay, a scene in the real world is viewed through a transmissivedisplay unit such as a half-silvered mirror, and a virtual object isprojected on the display unit.

The display device 30 including an image shooting device 35 (see FIG. 3)such as the smartphone described above is categorized as the VSTdisplay. The head-mounted display (HMD) described above is categorizedas the OST display because a scene in the real world is viewed throughlenses of the glasses serving as the display unit.

In the following embodiment, a mobile VST-display smartphone isexemplified as the display device 30 in FIG. 5. This smartphone may beowned by a visitor to the entertainment complex 10, or may be a leasableitem as typified by a tablet terminal to be leased to a visitor to theentertainment complex 10.

FIG. 3 exemplifies the hardware configuration of the display device 30.The display device 30 includes a central processing unit (CPU) 31, anacceleration sensor 32, a gyro sensor 33, a geomagnetic sensor 34, theimage shooting device 35, a GPS receiver 36, and the beacon receiver 37.The display device 30 further includes an input/output controller 39, asystem memory 40, a storage device 41, a graphics processing unit (GPU)42, a frame memory 43, a random access memory digital-to-analogconverter (RAMDAC) 44, a display controller 45, a display unit 46, andan inputter 47.

The system memory 40 is a storage to be used by an operating system (OS)executed by the CPU 31. The storage device 41 is an external storagethat stores, for example, a program for displaying a virtual realityimage (AR image) described later.

A direction and an azimuth of the display device 30 can be estimated byusing the acceleration sensor 32, the gyro sensor 33, and thegeomagnetic sensor 34. The acceleration sensor 32 measures anacceleration of the display device 30. As exemplified in FIG. 5, theacceleration sensor 32 can measure accelerations in three orthogonalaxial directions. That is, the acceleration sensor 32 measuresaccelerations in directions of an X axis and a Y axis orthogonal to eachother and parallel to a display surface of the display unit 46, and a Zaxis orthogonal to the display surface of the display unit 46. Theacceleration sensor 32, the gyro sensor 33, and the geomagnetic sensor34 are constructed from so-called micromachines such as Micro ElectroMechanical Systems (MEMS).

The gyro sensor 33 measures an angular velocity of the display device30. As exemplified in FIG. 5, the gyro sensor 33 measures rotationsabout the three orthogonal axes. That is, the gyro sensor 33 measures apitch angle corresponding to a rotation about the X axis, a roll anglecorresponding to a rotation about the Y axis, and an azimuth angle (alsoreferred to as “yaw angle”) corresponding to a rotation about the Zaxis. The geomagnetic sensor 34 detects an inclination of the displaydevice 30 from the magnetic north.

For example, the image shooting device 35 is a camera device mounted onthe smartphone, and can shoot a still or moving image of a scene in thereal world. The image shooting device 35 includes an image pickup devicesuch as a complementary metal oxide semiconductor (CMOS) or acharge-coupled device (CCD). The image shooting device 35 may be aso-called RGB-D camera having a function of measuring a separationdistance from the image shooting device 35 in addition to the functionof shooting the real world. For the function of measuring a separationdistance, the image shooting device 35 includes, for example, a distancemeasuring mechanism using an infrared ray in addition to the imagepickup device.

The GPU 42 is an image processor, and mainly operates for imagerecognition described later. The frame memory 43 is a storage thatstores an image shot by the image shooting device 35 and subjected toarithmetic processing by the GPU 42. The

RAMDAC 44 converts image data stored in the frame memory 43 into ananalog signal for the display unit 46 that is an analog display.

The GPS receiver 36 receives a GPS signal that is a positioning signalfrom a GPS satellite 24 (see FIG. 1). The GPS signal includes positioncoordinate information indicating a latitude, a longitude, and analtitude. The beacon receiver 37 receives a position signal output fromany beacon transmitter 22 in the entertainment complex 10.

The position estimating functions of the GPS receiver 36 and the beaconreceiver 37 overlap each other. Therefore, the display device 30 mayinclude only one of the GPS receiver 36 and the beacon receiver 37.

The inputter 47 can input an activation command and an image shootingcommand to the image shooting device 35. Examples of the inputter 47include a touch panel integrated with the display unit 46.

FIG. 6 exemplifies functional blocks of the display device 30. Forexample, the functional blocks are implemented such that the CPU 31 andthe GPU 42 execute programs stored in the system memory 40 and thestorage device 41. Alternatively, the functional blocks exemplified inFIG. 6 are implemented such that the CPU 31 and the GPU 42 executeprograms stored in a non-transitory computer-readable storage mediumsuch as a DVD or a hard disk drive of a computer.

FIG. 6 illustrates the configuration of the display device 30 in amixture of the functional blocks and a part of the hardwareconfiguration exemplified in FIG. 3. FIG. 6 exemplifies the imageshooting device 35, the display controller 45, the inputter 47, and thedisplay unit 46 as the hardware configuration.

The display device 30 includes a positional information acquirer 50, atransmitter 52, a receiver 55, a direction/azimuth estimator 56, and animage recognizer 58 as the functional blocks. The display device 30includes a virtual object data storage 57 and a trained model storage 59as storages. Those functional blocks are implemented by the CPU 31, thesystem memory 40, the storage device 41, the GPU 42, and the framememory 43.

The positional information acquirer 50 acquires information on a currentposition of the display device 30 from at least one of the GPS receiver36 and the beacon receiver 37 of FIG. 3. The positional information ispositional information in the so-called world coordinate system. In acase of a GPS signal, the positional information is included inpositional information in a space coordinate system, includinginformation on a latitude, a longitude, and an altitude. In a case wherethe received positional information is acquired from a beacon signal,the positional information in the world coordinate system includes, forexample, an x-coordinate, a y-coordinate, and a z-coordinate in thespace coordinate system having the origin at an arbitrary point in theentertainment complex 10. The x-coordinate and the y-coordinate may bedefined on orthogonal coordinate axes on a horizon plane, and thez-coordinate may be defined on a vertical axis.

The direction/azimuth estimator 56 estimates a direction and an azimuthof the display device 30. The direction/azimuth estimator 56 includesthe acceleration sensor 32 (see FIG. 3), the gyro sensor 33, and thegeomagnetic sensor 34. The geomagnetic sensor 34 can detect aninclination of the display device 30 from the magnetic north. Theacceleration sensor 32 and the gyro sensor 33 detect an inclination inthe Cartesian coordinate system illustrated in FIG. 5. For example, adirection of an optical axis of the image shooting device 35 relative tothe magnetic north can be determined.

The image recognizer 58 receives shot image data from the image shootingdevice 35 to execute image recognition. The image recognition includesrecognition of a real object in a shot image, and estimation of aseparation distance between the recognized real object and the displaydevice 30. For such image recognition, the shot image data includes, forexample, color image data obtained by shooting a scene in the realworld, and data on a separation distance between the image shootingdevice 35 and each pixel in the color image data.

The image recognizer 58 recognizes a real object in a shot image byusing a learning model for image recognition that is stored in thetrained model storage 59. For example, the trained model storage 59stores a neural network for image recognition that is trained by anexternal server. For example, data indicating an image of outdoorsincluding the entertainment complex 10 and subjected to segmentation andannotation of each object in the image is prepared as training data. Amulti-layer neural network is formed by supervised machine learningusing the training data, and is stored in the trained model storage 59.Examples of the neural network include a convolutional neural network(CNN).

As described above, the image shooting device 35 is the RGB-D camerahaving the distance measuring function, and can acquire a separationdistance between the image shooting device 35 and each pixel in a shotimage. A separation distance between the image shooting device 35 and areal object in the shot image that is recognized through the imagerecognition is determined by using the separation distance. Thus,positional information in a so-called local coordinate system isobtained for each real object in the shot image. The origin of the localcoordinate system is set to a predetermined point (for example, a focalpoint) on the image shooting device 35.

Based on the direction of the image shooting device 35 that isdetermined by the direction/azimuth estimator 56 and the position of thedisplay device 30 in the world coordinate system that is acquired by thepositional information acquirer 50, a position in the world coordinatesystem (position in the space coordinate system) can be determined forthe real object in the shot image.

Although the shot image shows a distant real object in a depthdirection, a fleer (character) near a chaser is generally captured inthe chasing game. For example, when distance information obtained foreach pixel in the shot image indicates a distance larger than apredetermined separation distance (for example, 100 m), the position inthe world coordinate system need not be determined for that pixel.

The virtual object data storage 57 stores image data of a virtual objectserving as a fleer, that is, a character. The virtual object datastorage 57 may store pieces of character image data in all the areas14A, 16A, and 18A. For example, the virtual object data storage 57stores one arbitrary type of character image data in each of the areas14A, 16A, and 18A. Each piece of character image data is associated withpositional information. For example, each piece of character image datais associated with a range of position coordinates in which the virtualobject can be displayed.

The display controller 45 can generate an augmented reality image (ARimage) by superimposing an image of a virtual object on a scene in thereal world, and cause the display unit 46 to display the AR image. Forexample, the display controller 45 superimposes an image of a characterthat is the virtual object on an image obtained by shooting the scene inthe real world with the image shooting device 35, and causes the displayunit 46 to display the superimposed image. Examples of the display unit46 include a liquid crystal display and an organic electroluminescence(EL) display.

As illustrated in FIG. 10, the display controller 45 generates anaugmented reality image by superimposing an image of a fleer virtualobject in the chasing game on an image shot by the image shooting device35. The display controller 45 acquires positional information(designated position) of the fleer (virtual object) from the server 70.The display controller 45 superimposes a character image that is theimage of the fleer at the designated position in the shot image. At thistime, a character image based on the designated position, that is,included in the designated position and set for the area 14A, 16A, or18A is selected as the character image to be superimposed.

Since a fleer (character) near a chaser is generally captured in thechasing game, the character image need not be superimposed on the shotimage for a fleer associated with a designated position at a distancelarger than a predetermined separation distance (for example, 100 m).

Configuration of Remote Terminal Device

The remote terminal device 100 is a manipulation device to bemanipulated by the manipulator 94 (see FIG. 1) at the place remote fromthe entertainment complex 10. Although FIG. 1 illustrates one remoteterminal device 100 and one manipulator 94, a plurality of remoteterminal devices 100 is communicable with the server 70, and therefore aplurality of manipulators 94 can participate in the chasing game asfleers.

Examples of the remote terminal device 100 (manipulation device) includea computer. Referring to FIG. 3, the remote terminal device 100 includesa CPU 101 serving as a processor, and a system memory 102 and a storagedevice 104 serving as storages.

The remote terminal device 100 includes a display unit 106, and adisplay controller 105 configured to control images to be displayed onthe display unit 106. The remote terminal device 100 includes aninput/output controller 103 configured to manage input and output ofinformation. The remote terminal device 100 is communicable with theserver 70 by a communication method such as the Internet.

The remote terminal device 100 includes an inputter 107 configured toinput information. Examples of the inputter 107 include a mouse and akeyboard that can input a manipulation command for a fleer. As describedlater, the manipulation command for the fleer includes a command to seta position (designated position) of a fleer virtual object.

FIG. 7 exemplifies functional blocks of the remote terminal device 100together with the hardware configuration in FIG. 3. The remote terminaldevice 100 includes a transmitter 110, a receiver 111, a virtual objectdata storage 112, and a facility map storage 113 as the functionalblocks.

The virtual object data storage 112 stores image data of a characterthat is a fleer virtual object. The character image data is associatedwith positional information. For example, a range of positioncoordinates in the entertainment complex 10 in which display of an imageof the fleer virtual object is permitted is stored in the virtual objectdata storage 112 in association with the character image data.

The facility map storage 113 stores park map data in the entertainmentcomplex 10. The park map data may be identical to the park map datastored in the facility map storage 81 of the server 70 (see FIG. 4).

The display controller 105 generates a game image by superimposingimages of fleers and chasers in the entertainment complex 10 on an imageof a park map, that is, a plane image of the entertainment complex 10 asillustrated in FIG. 14. For example, pieces of information on currentpositions of the fleers and the chasers are provided from the server 70.The display controller 105 superimposes the images of the fleers and thechasers on the park map at the current positions provided from theserver 70.

Flow on Chaser Side in Chasing Game

FIG. 8 exemplifies a flow on a chaser side in the chasing game servingas the AR game using the display system according to this embodiment. Inthe chaser-side flow, the server 70 and the display device 30 mainlyexecute steps. Among the steps of FIG. 8, steps represented by “(S)” areexecuted by the server 70. Steps represented by “(D)” are executed bythe display device 30.

Referring to FIG. 4, FIG. 6, and FIG. 8, the chaser flow is started whena request for participation registration in the chasing game is input tothe inputter 47 of the display device 30 in the entertainment complex10. In response to the input of the request for participationregistration, the positional information acquirer 50 of the displaydevice 30 acquires a current position of the display device 30. Thecurrent position is position coordinates in the world coordinate system.In response to the input of the request for participation registration,the image shooting device 35 is activated to shoot a scene in theentertainment complex 10.

The positional information acquired by the positional informationacquirer 50 and the participation registration information input fromthe inputter 47 are transmitted from the transmitter 52 to the server 70(S10). The receiver 83 of the server 70 transmits the positionalinformation and the participation registration information of thedisplay device 30 to the participant calculator 84. The receiver 83transmits the positional information of the display device 30 to the 3Dmodel extractor 86.

The 3D model extractor 86 extracts virtual object data from the themepark-specific character storage 82. As described above, the virtualobject data includes character image data and positional informationdata (display area data) associated with the character image data. Thevirtual object data is transmitted from the transmitter 87 to thedisplay device 30 (S12).

The participant calculator 84 stores the positional information and theparticipation registration information received from the receiver 83 inthe chasing game participant storage 80. The participant calculator 84acquires information on participants in the chasing game from thechasing game participant storage 80. This information includespositional information of a fleer. The positional information of thefleer may be positional information of every fleer participating in thechasing game. Alternatively, the positional information of the fleer maybe positional information of a fleer within a range of a predetermineddistance from the display device 30. For example, the participantcalculator 84 acquires positional information of a fleer within an areaof 100-m radius from the position of the display device 30. Thepositional information of the fleer may be position coordinateinformation in the world coordinate system. The participant calculator84 transmits the acquired positional information of the fleer to thedisplay device 30 (S14).

The image recognizer 58 of the display device 30 executes imagerecognition on an image shot by the image shooting device 35, andsegmentation and annotation on real objects in the shot image. Based onthe distance measuring function of the image shooting device 35, theimage recognizer 58 determines a separation distance between the displaydevice 30 and each recognized real object (S16).

Next, the image recognizer 58 causes the direction/azimuth estimator 56to determine a direction and an azimuth of the optical axis of the imageshooting device 35 (S18). For example, the image recognizer 58determines an azimuth (angle) of the optical axis of the image shootingdevice 35 relative to the magnetic north, and rotation angles(direction) about three orthogonal axes including the optical axis.

The image recognizer 58 causes the positional information acquirer 50 todetermine a position of the display device 30 in the world coordinatesystem (S20). The image recognizer 58 determines positions of the realobjects in the shot image in the world coordinate system (S22). The shotimage, the pieces of information on the real objects recognized in theimage, and the pieces of positional information of the real objects inthe world coordinate system are transmitted to the display controller45.

The display controller 45 determines position coordinates of fleers inthe shot image (S24). For example, the display controller 45 determineswhether position coordinates of a plurality of fleers that aretransmitted from the server 70 are in the shot image.

The display controller 45 superimposes character images that are virtualobject images at the positions (designated positions) of the fleers inthe shot image (S26). At this time, the character images to besuperimposed are selected based on the pieces of positional informationof the fleers. For example, character images set for the area 14A, 16A,or 18A including the positions of the fleers are selected andsuperimposed on the shot image.

FIG. 9 exemplifies a scene in the entertainment complex 10. FIG. 10illustrates an example in which the scene is shot by the image shootingdevice 35 of the display device 30 and the shot image is displayed onthe display unit 46. In this example, the shot image includes adesignated position (X1, Y1, Z1) of a fleer, and a character image 19 issuperimposed at the designated position. In this example, the designatedposition (X1, Y1, Z1) is in the zoo area 18A (see FIG. 2), and acharacter image 19 showing a monkey set for this area is superimposed onthe shot image.

The image recognizer 58 determines whether the virtual object iscaptured (S28). Referring to FIG. 11, a hand 120 of the chaser enters animage shooting field of the image shooting device 35 with the characterimage 19 displayed on the display unit 46. That is, the hand 120 entersan opposite side of the display unit 46.

The hand 120 of the chaser is shot by the image shooting device 35 andrecognized by the image recognizer 58. As exemplified in FIG. 12, whenthe position of a hand 121 of the chaser that is a real object in theshot image agrees with the designated position of the virtual object andthe hand 121 makes a predetermined gesture, the image recognizer 58recognizes the gesture. Through the gesture recognition, the imagerecognizer 58 determines that the fleer virtual object is captured bythe chaser as exemplified in FIG. 12. The gesture may be any action(capturing action) for capturing the character image that is the virtualobject image with the hand 120 on the display unit 46. When the imagerecognizer 58 recognizes the capturing action for the character image,the image recognizer 58 transmits a capture notification to the server70 via the transmitter 52 (S30). The capture notification includes anidentification symbol of the chaser (display device 30), the capturingposition, and an identification symbol of the captured fleer.

When the image recognizer 58 determines that the fleer is captured, thedisplay controller 45 executes image processing on the virtual objectwhen captured. As exemplified in FIG. 12, the display controller 45causes the display unit 46 to display a radial line image 122 indicatingthat the character image 19 has disappeared.

The capture notification is transmitted to the reward calculator 85 ofthe server 70. The reward calculator 85 gives a reward for the captureto the chaser (S32). For example, a score is transmitted to the displaydevice 30 as the reward. The score is displayed on the display unit 46of the display device 30. When the fleer is captured, the chasing gameof the chaser who has captured the fleer is terminated.

When the image recognizer 58 determines in Step S28 that the fleer(virtual object) is not captured by the chaser, the flow returns to StepS14 and the position coordinates of the fleers are acquired from theserver 70 again.

As described above, the chaser in the chasing game can enjoy the game ofchasing the virtual object (fleer) displayed on the display unit 46 ofthe display device 30.

The character image associated with the area 14A, 16A, or 18A where thevirtual object is located is selected as the image of the virtual objectto be displayed on the display unit 46. For example, the displayedcharacter image is changed when the position coordinates of the virtualobject moves beyond the area. This display control can provide arepresentation harmonized with each of the worlds of the theme parks 14,16, and 18 in the entertainment complex 10.

Flow on Fleer Side in Chasing Game

FIG. 13 exemplifies a flow on a fleer side in the chasing game. In thisflow, the server 70 and the remote terminal device 100 mainly executesteps. In FIG. 13, steps represented by “(S)” are executed by the server70, and steps represented by “(T)” are executed by the remote terminaldevice 100. Referring to FIG. 4, FIG. 7, and FIG. 13, the flow of FIG.13 (fleer-side flow) is started when a request for participationregistration in the chasing game is input to the inputter 107 of theremote terminal device 100.

The remote terminal device 100 transmits the participation registrationinformation to the server 70 (S50). The participation registrationinformation includes an identification symbol of the remote terminaldevice 100 or its manipulator 94 (see FIG. 1). The receiver 83 of theserver 70 that has received the participation registration informationtransmits the information to the participant calculator 84. Theparticipant calculator 84 stores the participation registrationinformation in the chasing game participant storage 80.

The participant calculator 84 acquires pieces of positional informationof all currently participating chasers from the chasing game participantstorage 80 (S52). Next, the participant calculator 84 sets an initialposition of the fleer near any chaser (S54). When a plurality of chasersis participating, any chaser is randomly selected and the initialposition of the fleer is set near the chaser. For example, the area nearthe chaser is a range of 1-m radius from the position of the chaser.

The participant calculator 84 acquires the park map data from thefacility map storage 81, and transmits the data to the remote terminaldevice 100 via the transmitter 87 (S56). The 3D model extractor 86receives the participation registration information, and transmitsvirtual object data to the remote terminal device 100.

Next, the participant calculator 84 transmits the pieces of positionalinformation of fleers and chasers to the remote terminal device 100(S58). The pieces of positional information of the chasers include thepieces of positional information of all the chasers participating in thechasing game in the entertainment complex 10. Similarly, the pieces ofpositional information of the fleers include pieces of positionalinformation of all the fleers participating in the chasing game in theentertainment complex 10. The positional information of the fleer alsoincludes the information on the initial position of the fleer that isset in Step S54, that is, the information on the initial position of thefleer of interest.

In the remote terminal device 100, the virtual object data storage 112stores the virtual object data. The facility map storage 113 stores thepark map data. The display controller 105 generates a park map image asa game image by superimposing virtual object images of the chasers andthe fleers including the fleer of interest, and causes the display unit106 to display the park map image (S60).

FIG. 14 exemplifies the game image. In the game image, character images15, 17, and 19 are superimposed on the park map image at the positioncoordinates of the fleers. The fleer character images 15, 17, and 19 areselected depending on which of the areas 14A, 16A, and 18A includes thepositions of the fleers. A chaser character image 125 may be setuniformly irrespective of the areas 14A, 16A, and 18A.

A character image 19A representing the fleer manipulated by themanipulator via the remote terminal device 100 may be subjected to imageprocessing such as hatching to discriminate from the other characterimages.

In response to the display of the game image on the display unit 106 asin FIG. 14, the manipulator 94 (see FIG. 1) inputs a manipulationcommand to the inputter 107 so that the manipulated fleer characterimage 19A can flee from the chasers (S62). That is, the manipulator 94inputs a manipulation command to the inputter 107 to set the designatedposition of the fleer (virtual object) to a position more distant fromthe chasers than the current position.

The input manipulation command is transmitted to the server 70. Thereward calculator 85 determines whether a capture notification about thecharacter image 19A is transmitted from the display device 30 of anychaser (S64). When the capture notification about the character image19A is not transmitted, the flow returns to Step S58.

When the capture notification about the character image 19A istransmitted to the server 70 in Step S64, the reward calculator 85calculates a reward.

First, the reward calculator 85 determines a score coefficient kl basedon a fleeing period of the character image 19A (S66). For example, thefleeing period is a period from a start timing of manipulation of thecharacter image 19A that is the virtual object to a timing ofrecognition of the capturing action, that is, a timing of output of thecapture notification. The score coefficient k1 is set to increase indirect proportion to the fleeing period.

Next, the reward calculator 85 determines a score coefficient k2 basedon a fleer density when the character image 19A is captured (S68). Thefleer density may be referred to as “virtual object space density”, andis a space density around the character image 19A that is the virtualobject image. For example, a space density of virtual objects (fleers)within a range of 5-m radius from the position coordinates of thecaptured character image 19A is calculated as the fleer density. A spacedensity of virtual objects (fleers) at the timing of recognition of thecapturing action, that is, the timing of output of the capturenotification is calculated as the fleer density.

As the fleer density increases, the fleer is less likely to be captured.Thus, the score coefficient k2 is set to decrease as the fleer densityincreases. For example, the fleer density and the score coefficient k2have an inversely proportional relationship.

Next, the reward calculator 85 determines a score coefficient k3 basedon a chaser density when the character image 19A is captured (S70). Thechaser density may be referred to as “chaser space density”, and is aspace density of chasers around the character image 19A. For example, aspace density of display devices 30 within a range of 5-m radius fromthe position coordinates of the character image 19A is calculated as thechaser density. A space density of display devices 30 at the timing ofrecognition of the capturing action, that is, the timing of output ofthe capture notification for the character image 19A is calculated asthe chaser density.

As the chaser density increases, the fleer is more likely to becaptured. Thus, the score coefficient k3 is set to increase as thechaser density increases. For example, the chaser density and the scorecoefficient k3 have a directly proportional relationship.

The reward calculator 85 calculates a score by multiplying apredetermined basic point by the score coefficients k1, k2, and k3, andgives the score to the remote terminal device 100 (S72). For example,the manipulator 94 (FIG. 1) who has manipulated the fleer via the remoteterminal device 100 can receive a discount service based on the givenscore when visiting the entertainment complex 10.

According to the display system described above, the fleeing route ofthe fleer in the chasing game is determined by the manipulator of theremote terminal device 100. Thus, it is possible to avoid a case wherethe fleeing route of the fleer is provided in the same pattern.

The manipulator can participate in the event of the entertainmentcomplex 10 at the place remote from the entertainment complex 10. Inaddition, the manipulator can receive the discount service when visitingthe entertainment complex 10. This service motivates the manipulator tomanipulate the fleer. Thus, a sufficient number of fleers can appear inthe chasing game.

The provider of the display system need not create an algorithm forsetting the fleeing route of the fleer, thereby reducing developmentcosts.

Other Example of Display System

FIG. 15 to FIG. 17 illustrate another example of the display systemaccording to this embodiment. In this system, the server 70 creates asuperimposed image by superimposing images of fleer virtual objects on ashot image in the real world. Further, the server 70 creates a gameimage by superimposing images of fleers and chasers on the park mapdata.

Referring to FIG. 15, the server 70 includes a display controller 88 inaddition to the functional blocks illustrated in FIG. 4. Functionalblocks overlapping those in FIG. 4 have the same functions, andtherefore description of those functional blocks is hereinafter omittedas appropriate. The display controller 88 has the functions of thedisplay controller 45 of the display device 30 (see FIG. 6) and thedisplay controller 105 of the remote terminal device 100 (see FIG. 7).That is, the display controller 88 generates an augmented reality imageto be displayed on the display device 30 and a game image to bedisplayed on the remote terminal device 100.

Along with the display controller 88 provided in the server 70, thedisplay controller 45 is omitted in the display device 30 as exemplifiedin FIG. 16. The virtual object data storage 57 is omitted as well. Inthe remote terminal device 100, the display controller 105, the virtualobject data storage 112, and the facility map storage 113 are omitted asexemplified in FIG. 17.

FIG. 18 exemplifies a chaser flow in the chasing game to be executed bythe server 70 and the display device 30 of FIG. 15 and FIG. 16. Detailsof processes in steps represented by the same reference symbols as thosein the chaser flow of FIG. 8 remain unchanged, and therefore descriptionof those steps is hereinafter omitted as appropriate.

In response to participation registration in the chasing game on thedisplay device 30, the display device 30 transmits participationregistration information to the server 70. This information alsoincludes positional information of the display device 30. Theparticipation registration information is stored in the chasing gameparticipant storage 80 of the server 70.

In Steps S16 to S22, real objects are recognized in a shot image, andpieces of positional information of the real objects in the worldcoordinate system are determined by the display device 30. Next, shotimage data with the pieces of positional information is transmitted tothe server 70 together with the positional information of the displaydevice 30 (S82). The display controller 88 of the server 70 determinesposition coordinates of fleers in the shot image (S84).

When the fleers are not captured by the chaser (that is, a personcarrying the display device 30), that is, the chasing game is still inprogress in Step S28, the display controller 88 superimposes characterimages of fleer virtual objects at the positions of the fleers in theshot image (S86). The character images to be superimposed are determinedbased on the positions of the fleers. For example, character images setin association with the area 14A, 16A, or 18A including the positions ofthe fleers are superimposed on the shot image. The superimposed image(AR image) is transmitted from the server 70 to the display device 30(S88).

FIG. 19 exemplifies a fleer flow in the chasing game to be executed bythe server 70 and the remote terminal device 100 of FIG. 15 and FIG. 17.Details of processes in steps represented by the same reference symbolsas those in the fleer flow of FIG. 13 remain unchanged, and thereforedescription of those steps is hereinafter omitted as appropriate.

When the participant calculator 84 of the server 70 acquires pieces ofpositional information of chasers and fleers from the chasing gameparticipant storage 80 in Step S58, the display controller 88 creates agame image by adding character images of the chasers and the fleers tothe park map image (S90). An identification image such as hatching isadded to the fleer associated with the remote terminal device 100 towhich the game image will be transmitted. The created game image istransmitted to the remote terminal device 100 (S92), and is displayed onthe display unit 106 of the remote terminal device 100.

According to the embodiment described above, the display controllers areomitted from the display device 30 and the remote terminal device 100.Thus, loads on those devices to create the augmented reality image andthe game image are reduced.

Other Example of Display Device

In the embodiment described above, the smartphone including the videosee-through display is exemplified as the display device 30. The displaydevice 30 according to this embodiment is not limited to this type ofdevice. As exemplified in FIG.

20, the display device 30 may include an optical see-through display asin a head-mounted display (HMD).

In this case, the display device 30 includes the image shooting device35, a half-silvered mirror 202 corresponding to the display unit 46, aprojector 203 corresponding to the display controller 45 and the imagerecognizer 58, and a sensor unit 201 corresponding to the positionalinformation acquirer 50.

For example, the half-silvered mirror 202 may be a lens of glasses orgoggles. Light (image) from the real world is transmitted to a wearerthrough the half-silvered mirror 202. The projector 203 arranged abovethe half-silvered mirror 202 projects an image of a virtual object onthe half-silvered mirror 202. Thus, an augmented reality image can bedisplayed by superimposing an image of a character that is a fleervirtual object on a scene in the entertainment complex 10 that is ascene in the real world.

What is claimed is:
 1. A display system comprising: a display deviceincluding: an image shooting device configured to shoot a scene in areal world; an image recognizer configured to recognize real objects inan image shot by the image shooting device, and recognize positions ofthe recognized real objects on space coordinates; and a displaycontroller configured to superimpose a virtual object at a designatedposition on the space coordinates in the shot image, the imagerecognizer being configured to recognize a gesture of the real object atthe designated position of the virtual object in the shot image, thedisplay controller being configured to execute image processing on thevirtual object in response to the recognized gesture; and a manipulationdevice including an inputter configured to input a manipulation commandto change the designated position of the virtual object.
 2. The displaysystem according to claim 1, further comprising a server communicablewith the display device and the manipulation device, wherein: thedisplay device is arranged in a complex facility including a pluralityof facilities with different specific themes; the server includes: astorage configured to store character images set as characters in theindividual facilities in association with pieces of positionalinformation in the complex facility; and a transmitter configured totransmit data on the character images to the display device togetherwith the pieces of positional information associated with the characterimages; and the display controller is configured to select, as an imageof the virtual object, the character image associated with thedesignated position, and superimpose the character image on the shotimage.
 3. The display system according to claim 2, wherein: the gestureis a capturing action at the designated position; the display device isconfigured to transmit a capture notification to the server when theimage recognizer recognizes the capturing action performed by the realobject; and the server includes a reward calculator configured todetermine a reward for a manipulator of the manipulation device based ona fleeing period from a start timing of manipulation of the virtualobject to a timing of recognition of the capturing action.
 4. Thedisplay system according to claim 3, wherein; the display deviceincludes an inputter configured to input participation registration as achaser in a chasing game for chasing the virtual object; themanipulation device includes an inputter configured to inputparticipation registration in the chasing game as a manipulator of thevirtual object; the server includes a participant calculator configuredto calculate the numbers of the chasers and the manipulatorsparticipating in the chasing game, and acquire positions in the complexfacility about the virtual object manipulated by the manipulator and thedisplay device of the chaser; and the reward calculator is configured todetermine the reward for the manipulator based on a virtual object spacedensity around the captured virtual object at the timing of recognitionof the capturing action.
 5. The display system according to claim 4,wherein the reward calculator is configured to determine the reward forthe manipulator of the manipulation device based on a chaser spacedensity around the captured virtual object at the timing of recognitionof the capturing action.
 6. A server communicable with display deviceseach configured to display a virtual object in a shot image obtained byshooting a scene in a real world and execute a chasing game for chasingthe virtual object, and with manipulation devices each including aninputter configured to input a manipulation command for the virtualobject, the server comprising: a storage configured to store pieces ofpositional information of the display devices participating in thechasing game, and pieces of positional information of the virtualobjects manipulated by the manipulation devices; and a transmitterconfigured to transmit, to a predetermined display device among thedisplay devices, the positional information of the virtual object aroundthe display device based on the positional information of the displaydevice.